JuliaSyntax parser-based REPL completions overhaul #57767

xal-0 · 2025-03-13T23:38:17Z

Overview

As we add REPL features, bugs related to the ad-hoc parsing done by
REPLCompletions.completions have crept in. This pull request replaces most of
the manual parsing (regex, find_start_brace) with a new approach that parses
the entire input buffer once, before and after the cursor, using JuliaSyntax.
We then query the parsed syntax tree to determine the kind of completion
to be done.

Changes

New, JuliaSyntax-based completions mechanism.
The complete_line interface now has the option of replacing arbitrary
regions of text in the input buffer by returning a Region (Pair{Int, Int}
for consistency with the convention in LineEdit, and pos being a 0-based
byte offset). This is used to improve the handling of auto-inserted closing
delimiters.. Leaving this unchanged for now.
Fixes parsing-related bugs:
Fixes some bugs that exist on 28d3bd5 that were found by fuzzing:
- x \" + TAB throws a FieldError exception
- String completion would sometimes delete the entire input buffer.
- Completions should not happen inside comments.
The duplicate code for path completion in strings, Cmd-strings, and the
shell has been removed, causing paths to complete the same way for all three.
Now, ~ is expanded in two situations:
- If foo exists, or if foo does not exist but there are no
  possible completions:
```
"~/foo/b|"     =TAB=>   "~/foo/bar|"
"~/foo/bar|"   =TAB=>   "/home/user/foo/bar|"
   OR
"~/foo/bar"|   =TAB=>   "/home/user/foo/bar"|
```
- If the current path ends with a / and you hit TAB again:
```
"~/foo/|"      =TAB=>   "/home/user/foo/|"
   OR
"~/foo/"|      =TAB=>   "/home/user/foo/"|
```

Future work

Method completions could be changed to look for methods with exactly the given
number of arguments if the closing ) is present, and search for signatures
with the right prefix otherwise.
- It would be nice to be able to search by type as well as value (perhaps
  by putting ::T in place of arguments).
Other REPL features could benefit from JuliaSyntax, so it might be worth
sharing the parse tree between completions and other features:
- Emacs-style sexpr navigation: C-M-f/C-M-b/C-M-u, etc.
- Improved auto-indent.
It would be nice if hints worked even when the cursor is between text.
CursorNode is a slightly tweaked copy of SyntaxNode from JuliaSyntax that
tracks the parent node but includes all trivia. It is used with seek_pos,
which navigates to the innermost node at a given position so we can examine
nearby nodes and the parent. This could probably duplicate less code from
JuliaSyntax.

Adds another permitted return type for complete_line, where the second element of the tuple is a Region (a Pair{Int, Int}) describing the region of text to be replaced. This is useful for making completions work consistently when the closing delimiter may or may not be present: the cursor can be made to "jump" out of the delimiters regardless of whether it is there already. "exam| =TAB=> "example.jl"| "exam|" =TAB=> "example.jl"|

This commit replaces the heuristic parsing done by REPLCompletions.completions with a new approach that parses the entire input buffer once with JuliaSyntax. In addition to fixing bugs, the more precise parsing should allow for new features in the future. Some features now work in more situations "for free", like dictionary key completion (the expression evaluated to find the keys is now more precise) and method suggestions (arguments beyond the cursor can be used to narrow the list). The tests have been updated to reflect slightly differing behaviour for string and Cmd-string completion: the new code returns a character range encompassing the entire string when completing paths (not observable by the user), and the behaviour of '~'-expansion has be tweaked to be consistent across all places where paths can be completed. Some escaping issues have also been fixed. Fixes: JuliaLang#55420, JuliaLang#55518, JuliaLang#55520, JuliaLang#55842, JuliaLang#56389, JuliaLang#57611

IanButterworth · 2025-03-14T02:43:30Z

This sounds great! I think we should backport it to at least 1.12 given it fixes so many issues.

giordano

@xal-0 first of all, speaking as the author of most of the issues linked to this PR, thank you for tackling this!

Also, can you please add a test for #57772 as well? I verified this works already (which is amazing!)

stdlib/REPL/test/replcompletions.jl

StefanKarpinski · 2025-03-14T15:25:17Z

Bravo. Truly excellent quality-of-life PR 👏🏻

giordano · 2025-03-14T17:40:12Z

Bravo. Truly excellent quality-of-life PR 👏🏻

Best part is that this PR has a negative diff, depsite the fact it added many tests.

stdlib/REPL/test/replcompletions.jl

…JuliaLang#57307, JuliaLang#57624 Also fix test failing for silly reason

Since we parse the entire input now, we need to avoid triggering method completion when the cursor is on the function name of a valid call.

stdlib/REPL/test/replcompletions.jl

IanButterworth · 2025-03-15T06:28:40Z

stdlib/REPL/src/REPLCompletions.jl

+
+    # Expand '~' if the user hits TAB after exhausting completions (either
+    # because we have found an existing file, or there is no such file).
+    full_path = ispath(path) || isempty(paths)


I found two cases that can cause ispath to error, so I think we should make a safe_ispath

ArgumentError: embedded NULs are not allowed in C strings: "invalid\0"
for let name = \"valid.txt\", name⁺ = \"invalid\\0.txt\"

IOError name too long (ENAMETOOLONG) for strings that are too long.

Something like this? Or just never throw and just return false?

function safe_ispath(path::AbstractString) try return ispath(path) catch err if err isa Base.IOError # i.e. name too long (ENAMETOOLONG) return false elseif err isa Base.ArgumentError && occursin("embedded NULs", err.msg) return false else rethrow() end end end

Maybe call it ispath_noerror or make it a keyword to ispath? The implication that ispath is unsafe or that this is "safer" feels unwarranted—it's just that there's situations where you don't want any errors.

I'll avoid modifying Base for the time being because we already convert some access(2) return values to false. It feels a little hacky to keep adding exceptions here.

julia/base/stat.jl

Lines 376 to 379 in bc98abc

r = ccall(:jl_fs_access, Cint, (Cstring, Cint), path, F_OK)

if !(r in (0, Base.UV_ENOENT, Base.UV_ENOTDIR, Base.UV_EINVAL))

uv_error(string("ispath(", repr(path), ")"), r)

end

Of the errors that could be returned for F_OK that we might want to handle, it looks like we're missing EACCES, EIO, ELOOP, and ENAMETOOLONG.

Co-authored-by: Ian Butterworth <[email protected]>

Co-authored-by: Mosè Giordano <[email protected]>

xal-0 · 2025-03-25T23:13:39Z

isdir suffers from similar problems as ispath. On Windows, isdir("D:\\") can fail if no disc is inserted, because libuv doesn't handle ERROR_NOT_READY:

julia> isdir("D:\\")
ERROR: IOError: stat("D:\\"): unknown error (UNKNOWN)
Stacktrace:
 [1] uv_error
   @ .\libuv.jl:106 [inlined]
 [2] checkstat
   @ .\stat.jl:186 [inlined]
 [3] stat(path::String)
   @ Base.Filesystem .\stat.jl:192
 [4] isdir(path::String)
   @ Base.Filesystem .\stat.jl:586
 [5] top-level scope
   @ REPL[1]:1

xal-0 · 2025-03-26T00:04:49Z

I am going to disable the shell completion tests until the shell mode can parse Windows paths...

shell> cd C:\Users
ERROR: IOError: cd("C:Users"): no such file or directory (ENOENT)
Stacktrace:
 [1] uv_error
   @ .\libuv.jl:106 [inlined]
 [2] cd(dir::String)
   @ Base.Filesystem .\file.jl:91
 [3] repl_cmd(cmd::Cmd, out::Base.TTY)
   @ Base .\client.jl:64
 [4] top-level scope
   @ none:1

Also cleans up do_cmd_escape, so that it can use different escaping syntax from the shell mode (which we may want to make similar to cmd.exe on Windows).

vtjnash · 2025-03-28T14:53:28Z

until the shell mode can parse Windows paths

The shell mode has no problem with Windows paths which is why the REPL has tests for it. It just sounds like you lost a call to Base.shell_escape on that code path.

vtjnash · 2025-03-28T15:05:51Z

stdlib/REPL/src/REPLCompletions.jl

-    for name in get_loading_candidates(pkgstarts, project_file)
-        push!(suggestions, PackageCompletion(name))
+function complete_loading_candidates!(suggestions::Vector{Completion}, s::String)
+    for name in ("Core", "Base")


I don't quite know the context of this, but should this be something like:

Suggested change

for name in ("Core", "Base")

for name in (r"Core\>", r"Base\>")

Since we have various modules like CoreLogging

Do you have an example where it misbehaves? This should only fire when we're completing the first component of an import path with no leading dots.

vtjnash · 2025-03-28T15:48:26Z

stdlib/REPL/src/REPLCompletions.jl

 end

-function shell_completions(string, pos, hint::Bool=false)
+function shell_completions(string, pos, hint::Bool=false; cmd_escape::Bool=false)


Since we always use the shell parser, why would we ever want this set true? I believe in the past, the code decided between whether it was completing one argument or several arguments, with escaping thus either wanting to include or exclude spaces. And thus choosing to fail if the input contained space-separate arguments rather than handling them, so I don't know why that would ever be useful.

vtjnash · 2025-03-28T16:01:35Z

stdlib/REPL/src/REPLCompletions.jl

-    # escape_raw_string with delim='`' and ignoring the rule for the ending \
-    return replace(s, r"(\\+)`" => s"\1\\`")
+function do_cmd_escape(s)
+    return Base.shell_escape_posixly(Base.escape_raw_string(s, '`'))


This seems like nonsense: I can't think of any case where we'd expect it'd be useful for the the REPL to pass the string unmodified to a posix shell for use as an argument in a julia script, which is what this transform order implements.

The previous transform took a string text that was already in a valid posix-shell form (e.g. for raw input to shell_parse for shell> text mode) and corrects it for the julia parser (e.g. for :(`text`)). Usually our shell_escape_posixly attempts to add quotes to avoid this edge case complexity, but if we have chosen to preserve the user-typed syntax then this is needed

julia> a = `\\ b\\\\` `'\' 'b\'`

I intended cmd_escape to make it possible to complete shell commands inside Cmd-strings, but mixed up the order (Base.escape_raw_string(Base.shell_escape_posixly(s), '`') does what I want I think):

julia> println(R.do_cmd_escape("file ` 1")) 'file \` 1' julia> @macroexpand `'file \` 1'` :(Base.cmd_gen((("file ` 1",),)))

Completion with shell_escape_posixly is still unwieldy because you get partial completions that insert only the opening quote, and can't complete any further.

stdlib/REPL/src/REPLCompletions.jl

Co-authored-by: Jameson Nash <[email protected]>

xal-0 added bugfix This change fixes an existing bug completions Tab and autocompletion in the repl labels Mar 13, 2025

xal-0 requested a review from vtjnash March 13, 2025 23:38

xal-0 force-pushed the juliasyntax-repl branch from 385cc7f to b62b012 Compare March 13, 2025 23:40

xal-0 mentioned this pull request Mar 13, 2025

REPL completions: Enter import mode only when cursor beyond "import" #57473

Closed

xal-0 added 2 commits March 13, 2025 16:49

xal-0 force-pushed the juliasyntax-repl branch from b62b012 to c539ec4 Compare March 13, 2025 23:49

xal-0 added the don't squash Don't squash merge label Mar 14, 2025

IanButterworth added the backport 1.12 Change should be backported to release-1.12 label Mar 14, 2025

giordano mentioned this pull request Mar 14, 2025

!ismi^tab does not tab complete inside function call #57772

Open

giordano reviewed Mar 14, 2025

View reviewed changes

stdlib/REPL/test/replcompletions.jl Outdated Show resolved Hide resolved

IanButterworth added the REPL Julia's REPL (Read Eval Print Loop) label Mar 14, 2025

xal-0 removed the don't squash Don't squash merge label Mar 14, 2025

giordano reviewed Mar 14, 2025

View reviewed changes

stdlib/REPL/test/replcompletions.jl Outdated Show resolved Hide resolved

xal-0 added 2 commits March 14, 2025 11:54

REPL: Add tests for JuliaLang#55420, JuliaLang#55429, JuliaLang#55842, …

304aefa

…JuliaLang#57307, JuliaLang#57624 Also fix test failing for silly reason

Add test for JuliaLang#57772

498aec3

xal-0 added 2 commits March 14, 2025 13:09

REPL: Add additional tests from JuliaLang#55518 discussion

5449069

REPL: Avoid triggering method completion when cursor on function name

a54d2c4

Since we parse the entire input now, we need to avoid triggering method completion when the cursor is on the function name of a valid call.

IanButterworth reviewed Mar 15, 2025

View reviewed changes

stdlib/REPL/test/replcompletions.jl Outdated Show resolved Hide resolved

IanButterworth mentioned this pull request Mar 15, 2025

REPL: Completion on a LOAD_PATH mutation breaks LOAD_PATH #57780

Closed

IanButterworth reviewed Mar 15, 2025

View reviewed changes

xal-0 and others added 4 commits March 17, 2025 16:49

Update stdlib/REPL/test/replcompletions.jl

3fad260

Co-authored-by: Ian Butterworth <[email protected]>

REPL: Catch IOError, ArgumentError when calling ispath

0b291a9

Co-authored-by: Ian Butterworth <[email protected]>

Update stdlib/REPL/test/replcompletions.jl

aef610e

Co-authored-by: Mosè Giordano <[email protected]>

Merge branch 'master' into juliasyntax-repl

50788f8

KristofferC mentioned this pull request Mar 20, 2025

Backports release 1.12 #57536

Merged

This was referenced Mar 22, 2025

REPL: know when to release the prefix context when completing #56322

Closed

REPL: complete imports when there are multiple macros before #56281

Closed

KristofferC mentioned this pull request Mar 24, 2025

Backports release 1.12 #57871

Merged

27 tasks

Merge branch 'master' into juliasyntax-repl

54b8f04

xal-0 added 3 commits March 25, 2025 17:32

REPL: Escape path completions after joining dir and path

9980020

Also cleans up do_cmd_escape, so that it can use different escaping syntax from the shell mode (which we may want to make similar to cmd.exe on Windows).

Merge remote-tracking branch 'upstream/master' into juliasyntax-repl

69db0a5

REPL: new backslash escape hack for Windows Pkg completions

d48fd5e

xal-0 force-pushed the juliasyntax-repl branch from 60dec58 to d48fd5e Compare March 26, 2025 19:48

vtjnash reviewed Mar 28, 2025

View reviewed changes

stdlib/REPL/src/REPLCompletions.jl Outdated Show resolved Hide resolved

KristofferC mentioned this pull request Mar 31, 2025

Backports release 1.12 #57955

Merged

36 tasks

Update stdlib/REPL/src/REPLCompletions.jl

8cd20d4

Co-authored-by: Jameson Nash <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JuliaSyntax parser-based REPL completions overhaul #57767

JuliaSyntax parser-based REPL completions overhaul #57767

xal-0 commented Mar 13, 2025 •

edited

Loading

IanButterworth commented Mar 14, 2025

giordano left a comment

StefanKarpinski commented Mar 14, 2025

giordano commented Mar 14, 2025

IanButterworth Mar 15, 2025 •

edited

Loading

StefanKarpinski Mar 16, 2025

xal-0 Mar 18, 2025

xal-0 commented Mar 25, 2025

xal-0 commented Mar 26, 2025

vtjnash commented Mar 28, 2025

vtjnash Mar 28, 2025

xal-0 Mar 31, 2025

vtjnash Mar 28, 2025

vtjnash Mar 28, 2025 •

edited

Loading

xal-0 Mar 31, 2025

xal-0 Mar 31, 2025

	r = ccall(:jl_fs_access, Cint, (Cstring, Cint), path, F_OK)
	if !(r in (0, Base.UV_ENOENT, Base.UV_ENOTDIR, Base.UV_EINVAL))
	uv_error(string("ispath(", repr(path), ")"), r)
	end

	for name in ("Core", "Base")
	for name in (r"Core\>", r"Base\>")

JuliaSyntax parser-based REPL completions overhaul #57767

Are you sure you want to change the base?

JuliaSyntax parser-based REPL completions overhaul #57767

Conversation

xal-0 commented Mar 13, 2025 • edited Loading

Overview

Changes

Future work

IanButterworth commented Mar 14, 2025

giordano left a comment

Choose a reason for hiding this comment

StefanKarpinski commented Mar 14, 2025

giordano commented Mar 14, 2025

IanButterworth Mar 15, 2025 • edited Loading

Choose a reason for hiding this comment

StefanKarpinski Mar 16, 2025

Choose a reason for hiding this comment

xal-0 Mar 18, 2025

Choose a reason for hiding this comment

xal-0 commented Mar 25, 2025

xal-0 commented Mar 26, 2025

vtjnash commented Mar 28, 2025

vtjnash Mar 28, 2025

Choose a reason for hiding this comment

xal-0 Mar 31, 2025

Choose a reason for hiding this comment

vtjnash Mar 28, 2025

Choose a reason for hiding this comment

vtjnash Mar 28, 2025 • edited Loading

Choose a reason for hiding this comment

xal-0 Mar 31, 2025

Choose a reason for hiding this comment

xal-0 Mar 31, 2025

Choose a reason for hiding this comment

xal-0 commented Mar 13, 2025 •

edited

Loading

IanButterworth Mar 15, 2025 •

edited

Loading

vtjnash Mar 28, 2025 •

edited

Loading